A growing amount of data generated in modern information systems presents considerable challenges with regard to storing, retaining, and managing information. These challenges have given rise to various data management technologies. For example, capacity planning, thin provisioning, and data reduction techniques are applied to improved efficiency in data storage systems. Data compression techniques have also been leveraged to address the magnitude of data stored by data storage systems.
Data de-duplication, also referred to as “de-dupe,” is another approach for improving capacity and efficiency in data storage systems. De-duplication is a data reduction technology that can compact a storage footprint by eliminating multiplicities, or copies, in the stored data. Since storage servers are often required to host files and data from multiple clients and users, many files or data elements may reside as multiple copies within the storage system. The copies may be in various seemingly unrelated folders. Even when each of these files is individually compressed, a great deal of efficiency may be obtained by eliminating the duplicated data elements. De-duplication at the file level can be implemented using hints obtained from file level meta-data to identify de-duplication candidate files. However, when dealing with unstructured data or with multiple versions of files that are different but share many blocks of common data, block level de-duplication may be more beneficial. Block level de-duplication may be far more difficult in environments where data is randomly accessed and altered after it has been de-duplicated.
Another concern in de-dupe implementations is related to having all of the eggs in one basket once multiple copies of a data element have been de-duplicated into one copy. Addressing data loss concerns in high performance block-level de-duplicating data storage systems introduces yet another challenge in implementing de-duplication techniques.
It is with respect to these considerations and others that the disclosure made herein is presented.